Stochastic Density Ratio Estimation and Its Application to Feature Selection

In this work, we deal with a relatively new statistical tool in machine learning: the estimation of the ratio of two probability densities, or density ratio estimation for short. As a side piece of research that gained its own traction, we also tackle the task of parameter selection in learning algorithms based on kernel methods. 1 Density Ratio Estimation The estimation of the ratio of two probability densities r(x) = p1(x) p2(x) is a statistical inference problem that finds useful applications in machine learning. Several approaches have been proposed and studied for the direct solution of the density ratio estimation problem, that is, to estimate the density ratio without going through density estimation [Sugiyama et al., 2011, and references therein]. By avoiding taking the ratio of two estimated densities, we avoid a dangerous source of error propagation. Next, we introduce situations where density ratio estimation naturally arises. Covariate-shift adaptation. Under the hood, most supervised learning algorithms apply the so-called Empirical Risk Minimization — ERM — principle, which selects a function f∗ n from a given set of functions F that minimizes the average of a loss function L : R × R 7→ R over a given set of training points {(x1, y1), . . . , (xn, yn)}. Formally: